{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "gpuType": "T4",
      "toc_visible": true,
      "authorship_tag": "ABX9TyNtyMzezZNzznP1h+a1anYg",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU",
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "36c5af36c7314efd9e62748975dcfe00": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_3e78773bbee644deb170a08e827b7c3a",
              "IPY_MODEL_3e4a031e2ac8488b9a09184ab6e05ab2",
              "IPY_MODEL_fd4b4560a97342978af87a11bbcaffed"
            ],
            "layout": "IPY_MODEL_5e7ac62f8a6b46b797770e6a48a83fca"
          }
        },
        "3e78773bbee644deb170a08e827b7c3a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_ba12aa09fa72474cb4f7e598ea38536b",
            "placeholder": "​",
            "style": "IPY_MODEL_8f0f021a566847cfb425f880f569fa0b",
            "value": "config.json: 100%"
          }
        },
        "3e4a031e2ac8488b9a09184ab6e05ab2": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_52e86180a03f4a9991e37d6b8f777957",
            "max": 843,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_96e1a19bfac349e199789876b76f85f5",
            "value": 843
          }
        },
        "fd4b4560a97342978af87a11bbcaffed": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_7664f9356fcb4fd4b5096cd0f1e19bca",
            "placeholder": "​",
            "style": "IPY_MODEL_8f0585c8ad264927a1eed30df76edcf4",
            "value": " 843/843 [00:00&lt;00:00, 43.1kB/s]"
          }
        },
        "5e7ac62f8a6b46b797770e6a48a83fca": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ba12aa09fa72474cb4f7e598ea38536b": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "8f0f021a566847cfb425f880f569fa0b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "52e86180a03f4a9991e37d6b8f777957": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "96e1a19bfac349e199789876b76f85f5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "7664f9356fcb4fd4b5096cd0f1e19bca": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "8f0585c8ad264927a1eed30df76edcf4": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "c3cf99bb27824b0fb13c8a15e776b90f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_11bdb8bcb9244f739bc86259a404bc1a",
              "IPY_MODEL_98e753dd8af24a93853a8f8e769b75bd",
              "IPY_MODEL_b5003a478cb44dfc9faf762e3dd5faf8"
            ],
            "layout": "IPY_MODEL_e1ff1913ecaf43b38a51bb72d12a7e33"
          }
        },
        "11bdb8bcb9244f739bc86259a404bc1a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_787154b843ce4d1e886f619e34e41ecc",
            "placeholder": "​",
            "style": "IPY_MODEL_2a1ecb799f58475ea95594457f22c442",
            "value": "model.safetensors: 100%"
          }
        },
        "98e753dd8af24a93853a8f8e769b75bd": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_0f41e1935110429ca92793d760169363",
            "max": 2471645608,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_09d0a0bc790d4952b404b0915a1099cb",
            "value": 2471645608
          }
        },
        "b5003a478cb44dfc9faf762e3dd5faf8": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_8f2c76e8ba7c4fbc8681ced2fdd214ff",
            "placeholder": "​",
            "style": "IPY_MODEL_bc89149c58654d628a2389291c74b8c1",
            "value": " 2.47G/2.47G [00:17&lt;00:00, 215MB/s]"
          }
        },
        "e1ff1913ecaf43b38a51bb72d12a7e33": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "787154b843ce4d1e886f619e34e41ecc": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2a1ecb799f58475ea95594457f22c442": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "0f41e1935110429ca92793d760169363": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "09d0a0bc790d4952b404b0915a1099cb": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "8f2c76e8ba7c4fbc8681ced2fdd214ff": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "bc89149c58654d628a2389291c74b8c1": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "e8cd2bc9fd324e15b8f747070e6b9838": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_2036d1f86ee848db9726b2628d560745",
              "IPY_MODEL_2b0514b0db5e4c97910c86136dab57b8",
              "IPY_MODEL_4b999b47c5fb4e889d0b201e53b25fa9"
            ],
            "layout": "IPY_MODEL_7c73e44eeb854dd19bae001685ac6041"
          }
        },
        "2036d1f86ee848db9726b2628d560745": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d4b8204a888b4e078a7d1e4270310209",
            "placeholder": "​",
            "style": "IPY_MODEL_38cfea66594343a08c964b341a30531e",
            "value": "generation_config.json: 100%"
          }
        },
        "2b0514b0db5e4c97910c86136dab57b8": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_a80961f5534145a5a10140d87c53cccb",
            "max": 185,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_3c3025f72cce4f9995c83c7360e632f6",
            "value": 185
          }
        },
        "4b999b47c5fb4e889d0b201e53b25fa9": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_7f8bf4422d5843b4b05fd398f5f5de72",
            "placeholder": "​",
            "style": "IPY_MODEL_b55f8a3d3a834abb8513f0ee3644491e",
            "value": " 185/185 [00:00&lt;00:00, 12.8kB/s]"
          }
        },
        "7c73e44eeb854dd19bae001685ac6041": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d4b8204a888b4e078a7d1e4270310209": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "38cfea66594343a08c964b341a30531e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "a80961f5534145a5a10140d87c53cccb": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "3c3025f72cce4f9995c83c7360e632f6": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "7f8bf4422d5843b4b05fd398f5f5de72": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b55f8a3d3a834abb8513f0ee3644491e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "70b3c73938df4feea10a4053c9625bec": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_38e9ddc8123a4b6f8030cfb64a64250b",
              "IPY_MODEL_f83566e82b5f4b93931c486edafd7cd1",
              "IPY_MODEL_dc8d72d552d4474e92cc7bf3311fac33"
            ],
            "layout": "IPY_MODEL_aa9f7b6a726e407cabcdecf91fc48263"
          }
        },
        "38e9ddc8123a4b6f8030cfb64a64250b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_a8a8063262104e418c5e5cf589a7622d",
            "placeholder": "​",
            "style": "IPY_MODEL_79b639863e8b4072b9a8995153fbac4c",
            "value": "tokenizer_config.json: 100%"
          }
        },
        "f83566e82b5f4b93931c486edafd7cd1": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d2c4b89293264189a1d4196f8dace9e9",
            "max": 50500,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_56b49dc4f4b047bf85438dd7236d1a2e",
            "value": 50500
          }
        },
        "dc8d72d552d4474e92cc7bf3311fac33": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_6a5606603c754e0d8ba55958a71c8a93",
            "placeholder": "​",
            "style": "IPY_MODEL_58348220f31c4cdca2a5cdbbc748c1fb",
            "value": " 50.5k/50.5k [00:00&lt;00:00, 3.33MB/s]"
          }
        },
        "aa9f7b6a726e407cabcdecf91fc48263": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "a8a8063262104e418c5e5cf589a7622d": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "79b639863e8b4072b9a8995153fbac4c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "d2c4b89293264189a1d4196f8dace9e9": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "56b49dc4f4b047bf85438dd7236d1a2e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "6a5606603c754e0d8ba55958a71c8a93": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "58348220f31c4cdca2a5cdbbc748c1fb": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "787439e60a5c41c2a3eb01242b88b26b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_ca79f147b3824bd1ac98aa6dcdc61302",
              "IPY_MODEL_094bc5941eb3465b88e2d54fce60cb91",
              "IPY_MODEL_ad01a05975af4c7faa787688d17697f9"
            ],
            "layout": "IPY_MODEL_7c82da21d43d45dfb65a14c277377435"
          }
        },
        "ca79f147b3824bd1ac98aa6dcdc61302": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_09953b7ada344b39baf816b5e99270cc",
            "placeholder": "​",
            "style": "IPY_MODEL_2bc69d1999b143258dd0093afdb98c49",
            "value": "tokenizer.json: 100%"
          }
        },
        "094bc5941eb3465b88e2d54fce60cb91": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_215a3c3a8b0d433faa57add94c7dd7a8",
            "max": 9085657,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_4c12ad74e646459c9058c996e36119fe",
            "value": 9085657
          }
        },
        "ad01a05975af4c7faa787688d17697f9": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_2246ecbeb4a0457dae337c2a17928718",
            "placeholder": "​",
            "style": "IPY_MODEL_2c025137106f4a4597d1d3e3bcda2805",
            "value": " 9.09M/9.09M [00:01&lt;00:00, 6.01MB/s]"
          }
        },
        "7c82da21d43d45dfb65a14c277377435": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "09953b7ada344b39baf816b5e99270cc": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2bc69d1999b143258dd0093afdb98c49": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "215a3c3a8b0d433faa57add94c7dd7a8": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "4c12ad74e646459c9058c996e36119fe": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "2246ecbeb4a0457dae337c2a17928718": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2c025137106f4a4597d1d3e3bcda2805": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "551fd3312fde4a7c9274ddf435f8c99a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_dcbdb9028d5746feb764cad2498a9a51",
              "IPY_MODEL_158fb255feb746749ed8a3ac4b850ea6",
              "IPY_MODEL_a0d7078f250241ef8eff9bd6622e3dca"
            ],
            "layout": "IPY_MODEL_7ca8566e3fb14a53982b5df43c7f07ac"
          }
        },
        "dcbdb9028d5746feb764cad2498a9a51": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_4d35768eb8fc44f49114125054286137",
            "placeholder": "​",
            "style": "IPY_MODEL_aa866a1b6c6b4155ab0359b7f6dd685b",
            "value": "special_tokens_map.json: 100%"
          }
        },
        "158fb255feb746749ed8a3ac4b850ea6": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_1fbac5fae0e34f5aae4a50becc51d637",
            "max": 301,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_d1a527a8d0334325b5954c584a5a2205",
            "value": 301
          }
        },
        "a0d7078f250241ef8eff9bd6622e3dca": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e37e5a13e37443dcb6fef6c061d11161",
            "placeholder": "​",
            "style": "IPY_MODEL_9f564828e4e04f34a7b5eaee2222779c",
            "value": " 301/301 [00:00&lt;00:00, 21.8kB/s]"
          }
        },
        "7ca8566e3fb14a53982b5df43c7f07ac": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "4d35768eb8fc44f49114125054286137": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "aa866a1b6c6b4155ab0359b7f6dd685b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "1fbac5fae0e34f5aae4a50becc51d637": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d1a527a8d0334325b5954c584a5a2205": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "e37e5a13e37443dcb6fef6c061d11161": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "9f564828e4e04f34a7b5eaee2222779c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "cece5695569b4532a42a0357874653f3": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_722a7745cc854eed957e6d0e0504ab9f",
              "IPY_MODEL_a222a90b5f4a4b2588ba94fabc213c6f",
              "IPY_MODEL_ba3630e5a35e419facbcfeffd2c6e3b5"
            ],
            "layout": "IPY_MODEL_d9501a0de69642a983a997440cc503e9"
          }
        },
        "722a7745cc854eed957e6d0e0504ab9f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_79e115b1505f45d593ca88f4918edb48",
            "placeholder": "​",
            "style": "IPY_MODEL_0f7935a9542e4e7fbcc3281b09d031d9",
            "value": "README.md: 100%"
          }
        },
        "a222a90b5f4a4b2588ba94fabc213c6f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d8c41ad13145411fb8d2f9f5022a9922",
            "max": 5174,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_ae5ab815572e4c139c0adc8cec2d6f3f",
            "value": 5174
          }
        },
        "ba3630e5a35e419facbcfeffd2c6e3b5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_275deb9c92f4446a9e8bcc10f6acc598",
            "placeholder": "​",
            "style": "IPY_MODEL_d83f182045164f0e9e3dc5c692135339",
            "value": " 5.17k/5.17k [00:00&lt;00:00, 330kB/s]"
          }
        },
        "d9501a0de69642a983a997440cc503e9": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "79e115b1505f45d593ca88f4918edb48": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "0f7935a9542e4e7fbcc3281b09d031d9": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "d8c41ad13145411fb8d2f9f5022a9922": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ae5ab815572e4c139c0adc8cec2d6f3f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "275deb9c92f4446a9e8bcc10f6acc598": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d83f182045164f0e9e3dc5c692135339": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "20a5b71d4bf941ecb0658807d499987b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_90c4ba92707f45d8913cea121aecb787",
              "IPY_MODEL_1953202045d44bfeb69c48c5539a7ef0",
              "IPY_MODEL_fb3e7a68c9314072bfbcd97d53c35501"
            ],
            "layout": "IPY_MODEL_db45ad2e736443439a9f9f831678ad96"
          }
        },
        "90c4ba92707f45d8913cea121aecb787": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d000d18a94954853a36989b06232a6e0",
            "placeholder": "​",
            "style": "IPY_MODEL_2a2d0e5bb41c444cb51709f6b87311d1",
            "value": "config.json: 100%"
          }
        },
        "1953202045d44bfeb69c48c5539a7ef0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_ccbcd291c3dd42ad8d403a306a72acaa",
            "max": 883,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_2a3f6d7d3d9f40afa0e0b0864207f0b2",
            "value": 883
          }
        },
        "fb3e7a68c9314072bfbcd97d53c35501": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_460b91727b4e40028886c84a1f2939d3",
            "placeholder": "​",
            "style": "IPY_MODEL_7c77de5450554cb3b660348ce5e49a0e",
            "value": " 883/883 [00:00&lt;00:00, 67.2kB/s]"
          }
        },
        "db45ad2e736443439a9f9f831678ad96": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d000d18a94954853a36989b06232a6e0": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2a2d0e5bb41c444cb51709f6b87311d1": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "ccbcd291c3dd42ad8d403a306a72acaa": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2a3f6d7d3d9f40afa0e0b0864207f0b2": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "460b91727b4e40028886c84a1f2939d3": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "7c77de5450554cb3b660348ce5e49a0e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "a559fdc00e2f418a9271507fce253084": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_d376c161c90041c4b52ba2280d4318d2",
              "IPY_MODEL_16a9ccc207e243fcbc0767bb074a6c0d",
              "IPY_MODEL_c55644804f1543339db4ca36496e605a"
            ],
            "layout": "IPY_MODEL_303a92373583484ab1a4b99adeccbdc2"
          }
        },
        "d376c161c90041c4b52ba2280d4318d2": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d469f4c9ba2c4d21bbe7b80f116bf6c3",
            "placeholder": "​",
            "style": "IPY_MODEL_d5c0cf067bad4b2f9bcfbd68599511a6",
            "value": "model.safetensors: 100%"
          }
        },
        "16a9ccc207e243fcbc0767bb074a6c0d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_b584051809f44f0282c066758b7f0b02",
            "max": 2149601544,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_8c14bb5122334c4c89df1057a2052bab",
            "value": 2149601544
          }
        },
        "c55644804f1543339db4ca36496e605a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_8309f0207309403eb8c352c6abc13b8f",
            "placeholder": "​",
            "style": "IPY_MODEL_e0a333ba28c1468398dee7e31b31ca03",
            "value": " 2.15G/2.15G [01:27&lt;00:00, 24.7MB/s]"
          }
        },
        "303a92373583484ab1a4b99adeccbdc2": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d469f4c9ba2c4d21bbe7b80f116bf6c3": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d5c0cf067bad4b2f9bcfbd68599511a6": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "b584051809f44f0282c066758b7f0b02": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "8c14bb5122334c4c89df1057a2052bab": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "8309f0207309403eb8c352c6abc13b8f": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e0a333ba28c1468398dee7e31b31ca03": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "58606e01c9974bd0aab0c4c1acf5622c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_213c320fd8e14840a47b29684e28bcf7",
              "IPY_MODEL_58cf861e60a7428fbf77946fa4305673",
              "IPY_MODEL_8c1665efbb83429fa656424c6d5fea0d"
            ],
            "layout": "IPY_MODEL_b31bb5958c6743f3a4a57112b42f4120"
          }
        },
        "213c320fd8e14840a47b29684e28bcf7": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_a077359ffe4643d99e8863f43dddcc45",
            "placeholder": "​",
            "style": "IPY_MODEL_20e3cfd7180e477d9ec300cc3d13f071",
            "value": "generation_config.json: 100%"
          }
        },
        "58cf861e60a7428fbf77946fa4305673": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_5cdb91d8ab904faeaa8085926286d999",
            "max": 180,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_76cf1894f25c4292ba893aa96a014f3c",
            "value": 180
          }
        },
        "8c1665efbb83429fa656424c6d5fea0d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_ed96a554823b403c9456def6de94e9db",
            "placeholder": "​",
            "style": "IPY_MODEL_adbfb2fed0f5429bb4c2f055f4c60200",
            "value": " 180/180 [00:00&lt;00:00, 13.9kB/s]"
          }
        },
        "b31bb5958c6743f3a4a57112b42f4120": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "a077359ffe4643d99e8863f43dddcc45": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "20e3cfd7180e477d9ec300cc3d13f071": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "5cdb91d8ab904faeaa8085926286d999": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "76cf1894f25c4292ba893aa96a014f3c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "ed96a554823b403c9456def6de94e9db": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "adbfb2fed0f5429bb4c2f055f4c60200": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "63774101422c4bfdb38142eb7715e1be": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_02c9cde98ece48b4adc2f20222b17ebb",
              "IPY_MODEL_f0e33cbe39f04935b155f908a5e75794",
              "IPY_MODEL_c45fb49784b645caa8dfa9411cddc1c7"
            ],
            "layout": "IPY_MODEL_a324b4e0bb6c422fb80fbb8e8e72172a"
          }
        },
        "02c9cde98ece48b4adc2f20222b17ebb": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_21dbb00986654628848fd8b287fcff9d",
            "placeholder": "​",
            "style": "IPY_MODEL_35ad7f0a261d4cfc847c36f8398809a7",
            "value": "tokenizer_config.json: 100%"
          }
        },
        "f0e33cbe39f04935b155f908a5e75794": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_f550089dca164f13943c12d6479c536f",
            "max": 50534,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_734dd78c1fb049ad94b743626016f06e",
            "value": 50534
          }
        },
        "c45fb49784b645caa8dfa9411cddc1c7": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_ac642c1ced614456b02a5f39b74c83fa",
            "placeholder": "​",
            "style": "IPY_MODEL_d488fd2e3d77411ca2a089e9f3492a26",
            "value": " 50.5k/50.5k [00:00&lt;00:00, 2.02MB/s]"
          }
        },
        "a324b4e0bb6c422fb80fbb8e8e72172a": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "21dbb00986654628848fd8b287fcff9d": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "35ad7f0a261d4cfc847c36f8398809a7": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "f550089dca164f13943c12d6479c536f": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "734dd78c1fb049ad94b743626016f06e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "ac642c1ced614456b02a5f39b74c83fa": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d488fd2e3d77411ca2a089e9f3492a26": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "b356ae4d6f7f4dc78daae9f00a338e1b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_8eb3c7f1f7794a17aa79843468b3926c",
              "IPY_MODEL_167b8ec730b046869bfeae60558eb9f8",
              "IPY_MODEL_eee14d76c00b4ba5a7129eae083ee33d"
            ],
            "layout": "IPY_MODEL_e29b9bdf5ca14faba8f13aeb4307deb8"
          }
        },
        "8eb3c7f1f7794a17aa79843468b3926c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_da0e1f16e0ec45d1acb710dcc9434bb9",
            "placeholder": "​",
            "style": "IPY_MODEL_f76d13949c1b41c9a7a48ef9349425f9",
            "value": "tokenizer.json: 100%"
          }
        },
        "167b8ec730b046869bfeae60558eb9f8": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_271b7314836d41a4947fc8749acffd76",
            "max": 9085657,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_83b4b4a046264ad3a156019bb76192b3",
            "value": 9085657
          }
        },
        "eee14d76c00b4ba5a7129eae083ee33d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_1b835b17d6fe4884b1a2e7e41bf988bb",
            "placeholder": "​",
            "style": "IPY_MODEL_ae13d551207c40b3a4ca2ce873df5f52",
            "value": " 9.09M/9.09M [00:01&lt;00:00, 8.30MB/s]"
          }
        },
        "e29b9bdf5ca14faba8f13aeb4307deb8": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "da0e1f16e0ec45d1acb710dcc9434bb9": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "f76d13949c1b41c9a7a48ef9349425f9": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "271b7314836d41a4947fc8749acffd76": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "83b4b4a046264ad3a156019bb76192b3": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "1b835b17d6fe4884b1a2e7e41bf988bb": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ae13d551207c40b3a4ca2ce873df5f52": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "89a298045c2d4a239ce0cad2f3737eeb": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_85d1c8ee730c40c9a8cac1fd152da378",
              "IPY_MODEL_93a87cd1cfd24110be3403f8138ddff8",
              "IPY_MODEL_11b0d4a003cb44fc8b7efef7022de0e0"
            ],
            "layout": "IPY_MODEL_e4a7e41649674af291674b3f20ef3772"
          }
        },
        "85d1c8ee730c40c9a8cac1fd152da378": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_777567f83624445cbc20bd9c965c0219",
            "placeholder": "​",
            "style": "IPY_MODEL_22cbf68fb2b348ecb5bcd1e622158727",
            "value": "special_tokens_map.json: 100%"
          }
        },
        "93a87cd1cfd24110be3403f8138ddff8": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_80a266d0cb1447e3a0a7e207063d76f3",
            "max": 335,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_bf15270bc3ff4344abc66a68be9ceac8",
            "value": 335
          }
        },
        "11b0d4a003cb44fc8b7efef7022de0e0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_c1e13124110844dd8ec749009492cbc3",
            "placeholder": "​",
            "style": "IPY_MODEL_c345ff9ff45140ba9a92974873178d75",
            "value": " 335/335 [00:00&lt;00:00, 24.3kB/s]"
          }
        },
        "e4a7e41649674af291674b3f20ef3772": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "777567f83624445cbc20bd9c965c0219": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "22cbf68fb2b348ecb5bcd1e622158727": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "80a266d0cb1447e3a0a7e207063d76f3": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "bf15270bc3ff4344abc66a68be9ceac8": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "c1e13124110844dd8ec749009492cbc3": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "c345ff9ff45140ba9a92974873178d75": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        }
      }
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/peremartra/Large-Language-Model-Notebooks-Course/blob/main/6-PRUNING/6_2_pruning_structured_llama3.2-1b_KO.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "<div>\n",
        "    <h1>Large Language Models Projects</a></h1>\n",
        "    <h3>Apply and Implement Strategies for Large Language Models</h3>\n",
        "    <h2>Pruning Llama 3.2.</h2>\n",
        "    <h3>Example of INCORRECT approach to pruning a Llama Model.</h3>\n",
        "</div>\n",
        "\n",
        "by [Pere Martra](https://www.linkedin.com/in/pere-martra/)\n",
        "_______\n",
        "Models: meta-llama/Llama-3.2-1B\n",
        "\n",
        "Colab Environment: GPU T4.\n",
        "\n",
        "Keys:\n",
        "* Pruning\n",
        "* Structured pruning\n",
        "\n",
        "\n",
        "Related article: --.\n",
        "_______\n",
        "**disclaimer: The pruning section was created after the first edition of the book was published. They are not included in the book’s original content but are intended to supplement and expand on the topics covered.**\n",
        "\n",
        "This is the unofficial repository for the book:\n",
        "        <a href=\"https://amzn.to/4eanT1g\"> <b>Large Language Models:</b> Apply and Implement Strategies for Large Language Models</a> (Apress).\n",
        "        The book is based on the content of this repository, but the notebooks are being updated, and I am incorporating new examples and chapters.\n",
        "        If you are looking for the official repository for the book, with the original notebooks, you should visit the\n",
        "        <a href=\"https://github.com/Apress/Large-Language-Models-Projects\">Apress repository</a>, where you can find all the notebooks in their original format as they appear in the book.\n",
        "______"
      ],
      "metadata": {
        "id": "hP9S5ckLF9lo"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Introduction\n",
        "This notebook cotinues the work done at: [6_1_pruning_structured_l1_diltilgpt2.ipynb](https://github.com/peremartra/Large-Language-Model-Notebooks-Course/blob/main/6-PRUNING/6_1_pruning_structured_l1_diltilgpt2.ipynb) where pruning was applied to a distilGPT2 model.\n",
        "\n",
        "The pruning process was based on selecting neurons from the model's feedforward layers that have the least importance using the L1 norm, assuming these contributed the least to the model's output.\n",
        "\n",
        "In this notebook, the same process is applied to a state-of-the-art model from the Llama family. The results, however, are not as expected, simply because the model's structure is very different, and the method needs to be adapted to these characteristics.\n",
        "\n",
        "**In this notebook, we'll identify the main issues so we can address them in a follow-up notebook.**"
      ],
      "metadata": {
        "id": "v4DPE3cPHIZC"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Install libraries & Configure variables."
      ],
      "metadata": {
        "id": "7d6zxPkdR88i"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Install necessary libraries.\n",
        "!pip install -q transformers\n",
        "!pip install -q torch\n",
        "!pip install -q sentencepiece  # Required for LLaMA tokenizer"
      ],
      "metadata": {
        "id": "5zHApVm41HWq"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#Import libraries\n",
        "import torch\n",
        "from transformers import AutoModelForCausalLM, AutoTokenizer\n",
        "from torch import nn\n",
        "import os\n",
        "\n",
        "# Check if GPU is available\n",
        "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
        "print(f\"Using device: {device}\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GJNgRj4M187E",
        "outputId": "52d3e6ba-6928-45da-b504-fbdaf5ff372a"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Using device: cuda\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Download model and explore structure"
      ],
      "metadata": {
        "id": "n0MedSz-SIYH"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model_name = 'meta-llama/Llama-3.2-1B'\n",
        "model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.float16).to(device)\n",
        "tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
        "tokenizer.pad_token = tokenizer.eos_token  # Set pad token"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 209,
          "referenced_widgets": [
            "36c5af36c7314efd9e62748975dcfe00",
            "3e78773bbee644deb170a08e827b7c3a",
            "3e4a031e2ac8488b9a09184ab6e05ab2",
            "fd4b4560a97342978af87a11bbcaffed",
            "5e7ac62f8a6b46b797770e6a48a83fca",
            "ba12aa09fa72474cb4f7e598ea38536b",
            "8f0f021a566847cfb425f880f569fa0b",
            "52e86180a03f4a9991e37d6b8f777957",
            "96e1a19bfac349e199789876b76f85f5",
            "7664f9356fcb4fd4b5096cd0f1e19bca",
            "8f0585c8ad264927a1eed30df76edcf4",
            "c3cf99bb27824b0fb13c8a15e776b90f",
            "11bdb8bcb9244f739bc86259a404bc1a",
            "98e753dd8af24a93853a8f8e769b75bd",
            "b5003a478cb44dfc9faf762e3dd5faf8",
            "e1ff1913ecaf43b38a51bb72d12a7e33",
            "787154b843ce4d1e886f619e34e41ecc",
            "2a1ecb799f58475ea95594457f22c442",
            "0f41e1935110429ca92793d760169363",
            "09d0a0bc790d4952b404b0915a1099cb",
            "8f2c76e8ba7c4fbc8681ced2fdd214ff",
            "bc89149c58654d628a2389291c74b8c1",
            "e8cd2bc9fd324e15b8f747070e6b9838",
            "2036d1f86ee848db9726b2628d560745",
            "2b0514b0db5e4c97910c86136dab57b8",
            "4b999b47c5fb4e889d0b201e53b25fa9",
            "7c73e44eeb854dd19bae001685ac6041",
            "d4b8204a888b4e078a7d1e4270310209",
            "38cfea66594343a08c964b341a30531e",
            "a80961f5534145a5a10140d87c53cccb",
            "3c3025f72cce4f9995c83c7360e632f6",
            "7f8bf4422d5843b4b05fd398f5f5de72",
            "b55f8a3d3a834abb8513f0ee3644491e",
            "70b3c73938df4feea10a4053c9625bec",
            "38e9ddc8123a4b6f8030cfb64a64250b",
            "f83566e82b5f4b93931c486edafd7cd1",
            "dc8d72d552d4474e92cc7bf3311fac33",
            "aa9f7b6a726e407cabcdecf91fc48263",
            "a8a8063262104e418c5e5cf589a7622d",
            "79b639863e8b4072b9a8995153fbac4c",
            "d2c4b89293264189a1d4196f8dace9e9",
            "56b49dc4f4b047bf85438dd7236d1a2e",
            "6a5606603c754e0d8ba55958a71c8a93",
            "58348220f31c4cdca2a5cdbbc748c1fb",
            "787439e60a5c41c2a3eb01242b88b26b",
            "ca79f147b3824bd1ac98aa6dcdc61302",
            "094bc5941eb3465b88e2d54fce60cb91",
            "ad01a05975af4c7faa787688d17697f9",
            "7c82da21d43d45dfb65a14c277377435",
            "09953b7ada344b39baf816b5e99270cc",
            "2bc69d1999b143258dd0093afdb98c49",
            "215a3c3a8b0d433faa57add94c7dd7a8",
            "4c12ad74e646459c9058c996e36119fe",
            "2246ecbeb4a0457dae337c2a17928718",
            "2c025137106f4a4597d1d3e3bcda2805",
            "551fd3312fde4a7c9274ddf435f8c99a",
            "dcbdb9028d5746feb764cad2498a9a51",
            "158fb255feb746749ed8a3ac4b850ea6",
            "a0d7078f250241ef8eff9bd6622e3dca",
            "7ca8566e3fb14a53982b5df43c7f07ac",
            "4d35768eb8fc44f49114125054286137",
            "aa866a1b6c6b4155ab0359b7f6dd685b",
            "1fbac5fae0e34f5aae4a50becc51d637",
            "d1a527a8d0334325b5954c584a5a2205",
            "e37e5a13e37443dcb6fef6c061d11161",
            "9f564828e4e04f34a7b5eaee2222779c"
          ]
        },
        "id": "q-z_1Zpg2I6u",
        "outputId": "ea77d99e-843f-4167-9f13-9c27fad751f8"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "config.json:   0%|          | 0.00/843 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "36c5af36c7314efd9e62748975dcfe00"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "model.safetensors:   0%|          | 0.00/2.47G [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "c3cf99bb27824b0fb13c8a15e776b90f"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "generation_config.json:   0%|          | 0.00/185 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "e8cd2bc9fd324e15b8f747070e6b9838"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "tokenizer_config.json:   0%|          | 0.00/50.5k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "70b3c73938df4feea10a4053c9625bec"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "tokenizer.json:   0%|          | 0.00/9.09M [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "787439e60a5c41c2a3eb01242b88b26b"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "special_tokens_map.json:   0%|          | 0.00/301 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "551fd3312fde4a7c9274ddf435f8c99a"
            }
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "def get_output(prompt, model=model, tokenizer=tokenizer):\n",
        "    inputs = tokenizer(prompt, return_tensors='pt').to(device)\n",
        "    outputs = model.generate(\n",
        "        inputs['input_ids'],\n",
        "        attention_mask=inputs['attention_mask'],\n",
        "        max_length=50,\n",
        "        num_return_sequences=1,\n",
        "        pad_token_id=tokenizer.pad_token_id\n",
        "    )\n",
        "    generated = tokenizer.decode(outputs[0], skip_special_tokens=True)\n",
        "    return generated"
      ],
      "metadata": {
        "id": "9UpMD4Hw2MWg"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## studying the model structure\n",
        "\n",
        "As you already know, studying the model’s structure is crucial for a successful pruning process.\n",
        "\n",
        "In this example, I’ll use the same pruning approach as in the previous example with a distilGPT2 model, which has a different structure. You can see the structure and the example in the notebook: [6_1_pruning_structured_l1_diltilgpt2.ipynb](https://github.com/peremartra/Large-Language-Model-Notebooks-Course/blob/main/6-PRUNING/6_1_pruning_structured_l1_diltilgpt2.ipynb).\n",
        "\n",
        "The process involved removing a percentage of the neurons with the lowest weights from the feedforward layers of the model, located within the MLP module. In the GPT2 model, these layers were called `c_fc` and `c_proj`, while in the Llama model, these layers are `gat_proj`, `up_proj`, and additionally `down_proj`.\n",
        "\n",
        "But the name isn’t the most important part, these layers have a very different structure and function compared to the `MLP` module layers in the distilGPT2 model.\n",
        "\n",
        "Understanding these differences will be crucial for defining the pruning process. In this notebook, we will examine how the Llama model  is negatively affected by a pruning process that worked correctly with the distilGPT2 model, even though both target the MLP layers and use the same neuron selection method.\n"
      ],
      "metadata": {
        "id": "m-3LoobgkA-x"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(model)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vkcJcUs6jLIn",
        "outputId": "3fb5960c-e90b-4db5-e854-e0f74f3d9d0c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "LlamaForCausalLM(\n",
            "  (model): LlamaModel(\n",
            "    (embed_tokens): Embedding(128256, 2048)\n",
            "    (layers): ModuleList(\n",
            "      (0-15): 16 x LlamaDecoderLayer(\n",
            "        (self_attn): LlamaSdpaAttention(\n",
            "          (q_proj): Linear(in_features=2048, out_features=2048, bias=False)\n",
            "          (k_proj): Linear(in_features=2048, out_features=512, bias=False)\n",
            "          (v_proj): Linear(in_features=2048, out_features=512, bias=False)\n",
            "          (o_proj): Linear(in_features=2048, out_features=2048, bias=False)\n",
            "          (rotary_emb): LlamaRotaryEmbedding()\n",
            "        )\n",
            "        (mlp): LlamaMLP(\n",
            "          (gate_proj): Linear(in_features=2048, out_features=8192, bias=False)\n",
            "          (up_proj): Linear(in_features=2048, out_features=8192, bias=False)\n",
            "          (down_proj): Linear(in_features=8192, out_features=2048, bias=False)\n",
            "          (act_fn): SiLU()\n",
            "        )\n",
            "        (input_layernorm): LlamaRMSNorm((2048,), eps=1e-05)\n",
            "        (post_attention_layernorm): LlamaRMSNorm((2048,), eps=1e-05)\n",
            "      )\n",
            "    )\n",
            "    (norm): LlamaRMSNorm((2048,), eps=1e-05)\n",
            "    (rotary_emb): LlamaRotaryEmbedding()\n",
            "  )\n",
            "  (lm_head): Linear(in_features=2048, out_features=128256, bias=False)\n",
            ")\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Each transformer block `lamaDecoderLayer` contains a MLP `LlamaMLP` Layer with a GLU (Gated Linear Structured.)\n",
        "\n",
        "It is a sophisticated structure whe comparing with other transformer models.\n",
        "\n",
        "Let's see each layer:\n",
        "* `gate_proj`: Projects the input to a higher dimension (2048 to 8192).\n",
        "\n",
        "* `up_proj`: Another projection to the higher dimension (2048 to 8192).\n",
        "\n",
        "* `down_proj`: Projects back to the original dimension (8192 to 2048).\n",
        "\n",
        "When prunig you should have in mind the relationship between these layers.\n",
        "\n",
        "Another important consideration is the model's configuration file. Since the pruning process alters the model's structure, the resulting structure must be reflected in the configuration file.\n",
        "\n",
        "Otherwise, we might encounter issues where the model doesn't work properly with the Transformers library or produces errors or incorrect results during inference.\n"
      ],
      "metadata": {
        "id": "WBIdtgJUEe1-"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(model.config)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "juOU8ij6Hhlv",
        "outputId": "705e2f35-b441-4243-cd15-3c8a9652d013"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "LlamaConfig {\n",
            "  \"_name_or_path\": \"meta-llama/Llama-3.2-1B\",\n",
            "  \"architectures\": [\n",
            "    \"LlamaForCausalLM\"\n",
            "  ],\n",
            "  \"attention_bias\": false,\n",
            "  \"attention_dropout\": 0.0,\n",
            "  \"bos_token_id\": 128000,\n",
            "  \"eos_token_id\": 128001,\n",
            "  \"head_dim\": 64,\n",
            "  \"hidden_act\": \"silu\",\n",
            "  \"hidden_size\": 2048,\n",
            "  \"initializer_range\": 0.02,\n",
            "  \"intermediate_size\": 8192,\n",
            "  \"max_position_embeddings\": 131072,\n",
            "  \"mlp_bias\": false,\n",
            "  \"model_type\": \"llama\",\n",
            "  \"num_attention_heads\": 32,\n",
            "  \"num_hidden_layers\": 16,\n",
            "  \"num_key_value_heads\": 8,\n",
            "  \"pretraining_tp\": 1,\n",
            "  \"rms_norm_eps\": 1e-05,\n",
            "  \"rope_scaling\": {\n",
            "    \"factor\": 32.0,\n",
            "    \"high_freq_factor\": 4.0,\n",
            "    \"low_freq_factor\": 1.0,\n",
            "    \"original_max_position_embeddings\": 8192,\n",
            "    \"rope_type\": \"llama3\"\n",
            "  },\n",
            "  \"rope_theta\": 500000.0,\n",
            "  \"tie_word_embeddings\": true,\n",
            "  \"torch_dtype\": \"float16\",\n",
            "  \"transformers_version\": \"4.44.2\",\n",
            "  \"use_cache\": true,\n",
            "  \"vocab_size\": 128256\n",
            "}\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Test the original model\n",
        "prompt = \"Paris is the capital of\"\n",
        "generated = get_output(prompt)\n",
        "print(f\"Generated text: {generated}\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QCxjsR4nSP31",
        "outputId": "18c88dec-37ba-43a9-e9dd-9e2ae1c7a574"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Generated text: Paris is the capital of France and the most popular tourist destination in the country. It’s also the largest city in Europe, with a population of more than 2 million people. Paris is a city of culture, art, and history, and\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Support function to check the size reduction.\n",
        "def count_parameters(model):\n",
        "    return sum(p.numel() for p in model.parameters())"
      ],
      "metadata": {
        "id": "8WR96iwq2XYH"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "original_param_count = count_parameters(model)\n",
        "print(f\"Original model parameters: {original_param_count}\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YiZf0g9pQvbD",
        "outputId": "938fa64e-1360-4101-f1f7-069672da5b9b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Original model parameters: 1235814400\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Pruning Model.\n",
        "\n",
        "## Support pruning functions."
      ],
      "metadata": {
        "id": "LN5-UakQJOPU"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Function to compute importance scores (L1 norm)\n",
        "def compute_importance_scores(layer_weight):\n",
        "    \"\"\"\n",
        "    compute importance scores (L1 norm)\n",
        "\n",
        "    Args:\n",
        "    - layer_weight: Weight matrix from a gate_proj / up_proj layer.\n",
        "\n",
        "    Returns:\n",
        "    - importance_scores: L1 norm Importance scores for each neuron.\n",
        "    \"\"\"\n",
        "    weight = layer_weight.float()\n",
        "    return torch.sum(torch.abs(weight), dim=1)"
      ],
      "metadata": {
        "id": "_mMGhTtD4PRX"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def prune_neurons(mlp, prune_percent):\n",
        "    \"\"\"\n",
        "    Prune neurons from the gate_weight and c_proj layers of the MLP based on importance scores.\n",
        "\n",
        "    Args:\n",
        "    - mlp: The MLP layer (contains gate_proj and up_proj) to prune.\n",
        "    - prune_percent: Percentage of neurons to prune.\n",
        "\n",
        "    Returns:\n",
        "    - new_gate_proj: New pruned c_fc layer.\n",
        "    - new_up_proj: New pruned c_proj layer.\n",
        "    - new_down_proj: New down_proj layer.\n",
        "    - new_size: Number of neurons after pruning.\n",
        "    - indices_to_keep: Indices of neurons to keep.\n",
        "    \"\"\"\n",
        "    # Get the weights of the gate_proj and up_proj layers\n",
        "    gate_weight = mlp.gate_proj.weight.data.float()  # Shape: [output_features, input_features]\n",
        "    up_weight = mlp.up_proj.weight.data.float()      # Shape: [output_features, input_features]\n",
        "\n",
        "    print(f\"gate_weight.shape: {gate_weight.shape}\")\n",
        "    print(f\"up_weight.shape: {up_weight.shape}\")\n",
        "\n",
        "    # Compute importance scores for each neuron separately and sum them\n",
        "    importance_scores_gate = compute_importance_scores(gate_weight)\n",
        "    importance_scores_up = compute_importance_scores(up_weight)\n",
        "    importance_scores = importance_scores_gate + importance_scores_up\n",
        "\n",
        "    # Check for NaNs or Infs\n",
        "    if torch.isnan(importance_scores).any():\n",
        "        print(\"Warning: importance_scores contains NaNs\")\n",
        "    if torch.isinf(importance_scores).any():\n",
        "        print(\"Warning: importance_scores contains Infs\")\n",
        "\n",
        "    # Determine the number of neurons to prune\n",
        "    original_intermediate_size = gate_weight.size(0)  # This is output_features\n",
        "    num_neurons_to_prune = int(prune_percent * original_intermediate_size)\n",
        "\n",
        "    # Ensure num_neurons_to_prune is valid\n",
        "    num_neurons_to_prune = max(0, min(num_neurons_to_prune, original_intermediate_size - 1))\n",
        "    k = original_intermediate_size - num_neurons_to_prune\n",
        "\n",
        "    print(f\"Original intermediate size: {original_intermediate_size}\")\n",
        "    print(f\"Number of neurons to prune: {num_neurons_to_prune}\")\n",
        "    print(f\"Number of neurons to keep (k): {k}\")\n",
        "\n",
        "    if k <= 0:\n",
        "        raise ValueError(f\"Invalid number of neurons to keep: {k}. Adjust the prune_percent or check the layer sizes.\")\n",
        "\n",
        "    # Ensure importance_scores is on the same device\n",
        "    importance_scores = importance_scores.to(device)\n",
        "\n",
        "    # Get indices of neurons to keep (those with highest importance)\n",
        "    _, indices_to_keep = torch.topk(importance_scores, k)\n",
        "\n",
        "    # Sort indices to maintain order\n",
        "    indices_to_keep, _ = torch.sort(indices_to_keep)\n",
        "\n",
        "    # Create new Linear layers with reduced size\n",
        "    new_gate_proj = nn.Linear(mlp.gate_proj.in_features, len(indices_to_keep), bias=False).to(device)\n",
        "    new_up_proj = nn.Linear(mlp.up_proj.in_features, len(indices_to_keep), bias=False).to(device)\n",
        "    new_down_proj = nn.Linear(len(indices_to_keep), mlp.down_proj.out_features, bias=False).to(device)\n",
        "\n",
        "    return new_gate_proj, new_up_proj, new_down_proj, len(indices_to_keep), indices_to_keep\n"
      ],
      "metadata": {
        "id": "NX9Boph94RWA"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Function to copy weights and biases to new pruned layers\n",
        "def copy_weights_and_biases(mlp, new_gate_proj, new_up_proj, new_down_proj, indices_to_keep):\n",
        "    \"\"\"\n",
        "    Copy the weights and biases from the original layers to the new pruned layers.\n",
        "\n",
        "    Args:\n",
        "    - mlp: The original MLP layer.\n",
        "    - new_cnew_gate_proj_fc: New pruned gate_proj layer.\n",
        "    - new_up_proj: New pruned up_proj layer.\n",
        "    - new_down_proj: New pruned down_proj layer.\n",
        "    - indices_to_keep: Indices of neurons that are retained.\n",
        "    \"\"\"\n",
        "    # Copy weights for gate_proj and up_proj (input features remain the same)\n",
        "    new_gate_proj.weight.data = mlp.gate_proj.weight.data[indices_to_keep, :]\n",
        "    new_up_proj.weight.data = mlp.up_proj.weight.data[indices_to_keep, :]\n",
        "\n",
        "    # Copy weights for down_proj (output features remain the same)\n",
        "    new_down_proj.weight.data = mlp.down_proj.weight.data[:, indices_to_keep]\n"
      ],
      "metadata": {
        "id": "FxJEWg1X3j0m"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Prune Loop\n",
        "The update_model function iterates through the blocks within the model's Transformer structure. This structure consists of multiple `LlamaDecoderLayer` blocks, and each of these blocks contains a pair of `LlamaSdpaAttention` and `LlamaMLP` components. The latter contains the MLP layers that will be the target of the pruning process.\n",
        "\n",
        "```\n",
        "(layers): ModuleList(\n",
        "      (0-15): 16 x LlamaDecoderLayer(\n",
        "        (self_attn): LlamaSdpaAttention(\n",
        "          (q_proj): Linear(in_features=2048, out_features=2048, bias=False)\n",
        "          (k_proj): Linear(in_features=2048, out_features=512, bias=False)\n",
        "          (v_proj): Linear(in_features=2048, out_features=512, bias=False)\n",
        "          (o_proj): Linear(in_features=2048, out_features=2048, bias=False)\n",
        "          (rotary_emb): LlamaRotaryEmbedding()\n",
        "        )\n",
        "        (mlp): LlamaMLP(\n",
        "          (gate_proj): Linear(in_features=2048, out_features=8192, bias=False)\n",
        "          (up_proj): Linear(in_features=2048, out_features=8192, bias=False)\n",
        "          (down_proj): Linear(in_features=8192, out_features=2048, bias=False)\n",
        "          (act_fn): SiLU()\n",
        "        )\n",
        "        (input_layernorm): LlamaRMSNorm((2048,), eps=1e-05)\n",
        "        (post_attention_layernorm): LlamaRMSNorm((2048,), eps=1e-05)\n",
        "      )\n",
        "  )    \n",
        "```\n",
        "\n",
        "The layers that will undergo the removal of neurons identified as less useful are:\n",
        "```\n",
        "(gate_proj): Linear(in_features=2048, out_features=8192, bias=False)\n",
        "(up_proj): Linear(in_features=2048, out_features=8192, bias=False)\n",
        "(down_proj): Linear(in_features=8192, out_features=2048, bias=False)\n",
        "```\n",
        "The neurons are removed in the `prune_neurons` function based on the values returned by `compute_importance_scores`.\n",
        "\n"
      ],
      "metadata": {
        "id": "C3s_MoQdT40P"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Function to update the model\n",
        "def update_model(model, prune_percent):\n",
        "    new_intermediate_size = None\n",
        "\n",
        "    for idx, layer in enumerate(model.model.layers):\n",
        "        mlp = layer.mlp\n",
        "\n",
        "        # Prune the neurons and create new layers\n",
        "        new_gate_proj, new_up_proj, new_down_proj, new_size, indices_to_keep = prune_neurons(mlp, prune_percent)\n",
        "\n",
        "        # Copy weights from old layers to new pruned layers\n",
        "        copy_weights_and_biases(mlp, new_gate_proj, new_up_proj, new_down_proj, indices_to_keep)\n",
        "\n",
        "        # Replace old layers with new pruned layers\n",
        "        mlp.gate_proj = new_gate_proj\n",
        "        mlp.up_proj = new_up_proj\n",
        "        mlp.down_proj = new_down_proj\n",
        "\n",
        "        # Update the intermediate size for the first layer\n",
        "        if new_intermediate_size is None:\n",
        "            new_intermediate_size = new_size\n",
        "\n",
        "    # Update the model configuration with the new intermediate size\n",
        "    model.config.intermediate_size = new_intermediate_size\n",
        "\n",
        "    return model"
      ],
      "metadata": {
        "id": "CxblrrCXPzEt"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Obtain & test the model.  "
      ],
      "metadata": {
        "id": "VILv7TS5P4XJ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "prune_percent = 0.2  # Prune 20% of neurons\n",
        "model = update_model(model, prune_percent)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NIUnFU5R3n42",
        "outputId": "c1395964-f047-46b1-ef1e-c9496f0c3459"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "gate_weight.shape: torch.Size([8192, 2048])\n",
            "up_weight.shape: torch.Size([8192, 2048])\n",
            "Original intermediate size: 8192\n",
            "Number of neurons to prune: 1638\n",
            "Number of neurons to keep (k): 6554\n",
            "gate_weight.shape: torch.Size([8192, 2048])\n",
            "up_weight.shape: torch.Size([8192, 2048])\n",
            "Original intermediate size: 8192\n",
            "Number of neurons to prune: 1638\n",
            "Number of neurons to keep (k): 6554\n",
            "gate_weight.shape: torch.Size([8192, 2048])\n",
            "up_weight.shape: torch.Size([8192, 2048])\n",
            "Original intermediate size: 8192\n",
            "Number of neurons to prune: 1638\n",
            "Number of neurons to keep (k): 6554\n",
            "gate_weight.shape: torch.Size([8192, 2048])\n",
            "up_weight.shape: torch.Size([8192, 2048])\n",
            "Original intermediate size: 8192\n",
            "Number of neurons to prune: 1638\n",
            "Number of neurons to keep (k): 6554\n",
            "gate_weight.shape: torch.Size([8192, 2048])\n",
            "up_weight.shape: torch.Size([8192, 2048])\n",
            "Original intermediate size: 8192\n",
            "Number of neurons to prune: 1638\n",
            "Number of neurons to keep (k): 6554\n",
            "gate_weight.shape: torch.Size([8192, 2048])\n",
            "up_weight.shape: torch.Size([8192, 2048])\n",
            "Original intermediate size: 8192\n",
            "Number of neurons to prune: 1638\n",
            "Number of neurons to keep (k): 6554\n",
            "gate_weight.shape: torch.Size([8192, 2048])\n",
            "up_weight.shape: torch.Size([8192, 2048])\n",
            "Original intermediate size: 8192\n",
            "Number of neurons to prune: 1638\n",
            "Number of neurons to keep (k): 6554\n",
            "gate_weight.shape: torch.Size([8192, 2048])\n",
            "up_weight.shape: torch.Size([8192, 2048])\n",
            "Original intermediate size: 8192\n",
            "Number of neurons to prune: 1638\n",
            "Number of neurons to keep (k): 6554\n",
            "gate_weight.shape: torch.Size([8192, 2048])\n",
            "up_weight.shape: torch.Size([8192, 2048])\n",
            "Original intermediate size: 8192\n",
            "Number of neurons to prune: 1638\n",
            "Number of neurons to keep (k): 6554\n",
            "gate_weight.shape: torch.Size([8192, 2048])\n",
            "up_weight.shape: torch.Size([8192, 2048])\n",
            "Original intermediate size: 8192\n",
            "Number of neurons to prune: 1638\n",
            "Number of neurons to keep (k): 6554\n",
            "gate_weight.shape: torch.Size([8192, 2048])\n",
            "up_weight.shape: torch.Size([8192, 2048])\n",
            "Original intermediate size: 8192\n",
            "Number of neurons to prune: 1638\n",
            "Number of neurons to keep (k): 6554\n",
            "gate_weight.shape: torch.Size([8192, 2048])\n",
            "up_weight.shape: torch.Size([8192, 2048])\n",
            "Original intermediate size: 8192\n",
            "Number of neurons to prune: 1638\n",
            "Number of neurons to keep (k): 6554\n",
            "gate_weight.shape: torch.Size([8192, 2048])\n",
            "up_weight.shape: torch.Size([8192, 2048])\n",
            "Original intermediate size: 8192\n",
            "Number of neurons to prune: 1638\n",
            "Number of neurons to keep (k): 6554\n",
            "gate_weight.shape: torch.Size([8192, 2048])\n",
            "up_weight.shape: torch.Size([8192, 2048])\n",
            "Original intermediate size: 8192\n",
            "Number of neurons to prune: 1638\n",
            "Number of neurons to keep (k): 6554\n",
            "gate_weight.shape: torch.Size([8192, 2048])\n",
            "up_weight.shape: torch.Size([8192, 2048])\n",
            "Original intermediate size: 8192\n",
            "Number of neurons to prune: 1638\n",
            "Number of neurons to keep (k): 6554\n",
            "gate_weight.shape: torch.Size([8192, 2048])\n",
            "up_weight.shape: torch.Size([8192, 2048])\n",
            "Original intermediate size: 8192\n",
            "Number of neurons to prune: 1638\n",
            "Number of neurons to keep (k): 6554\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "As is posible to see in this simple log we are reducing the number of features in the upgrad layers from 8192 to 6554. Ther are 16 * 2 layers affected by the reduction."
      ],
      "metadata": {
        "id": "sSWBiV9-YpL5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Recalculate the number of parameters\n",
        "pruned_param_count = count_parameters(model)\n",
        "reduction_in_params = original_param_count - pruned_param_count\n",
        "percentage_savings = (reduction_in_params / original_param_count) * 100\n",
        "\n",
        "print(f\"Pruned model parameters: {pruned_param_count}\")\n",
        "print(f\"Reduction in parameters: {reduction_in_params}\")\n",
        "print(f\"Percentage of weight savings: {percentage_savings:.2f}%\")\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tdJUkfWI3qMM",
        "outputId": "03030785-a90f-476d-8a9a-75594aed9db0"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Pruned model parameters: 1074792448\n",
            "Reduction in parameters: 161021952\n",
            "Percentage of weight savings: 13.03%\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Test the pruned model\n",
        "generated = get_output(prompt, model, tokenizer)\n",
        "print(f\"Generated text after pruning: {generated}\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wvj-iIsO5M6U",
        "outputId": "db6b53e1-6a80-44f4-e140-720ec6e163df"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Generated text after pruning: Paris is the capital of of France. This is the the the the main the area of. This is the the the the the the the the the the the the the the the the city of the the France of the of the of the of\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**-- WARNING --**\n",
        "\n",
        "Although it's normal for a model to lose some capabilities due to a pruning process, what has happened to our model is not normal.\n",
        "\n",
        "It's not just a matter of reducing the pruning percentage. The issue here runs deeper. There are a couple of problems in the pruning process that need to be addressed."
      ],
      "metadata": {
        "id": "200MPWCnZGKC"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Identifying the problems.\n"
      ],
      "metadata": {
        "id": "D0ZJObyYi55q"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(model)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_hNaWinNXkhz",
        "outputId": "0aec3c75-b288-453c-9d3a-ed4f04d05cd6"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "LlamaForCausalLM(\n",
            "  (model): LlamaModel(\n",
            "    (embed_tokens): Embedding(128256, 2048)\n",
            "    (layers): ModuleList(\n",
            "      (0-15): 16 x LlamaDecoderLayer(\n",
            "        (self_attn): LlamaSdpaAttention(\n",
            "          (q_proj): Linear(in_features=2048, out_features=2048, bias=False)\n",
            "          (k_proj): Linear(in_features=2048, out_features=512, bias=False)\n",
            "          (v_proj): Linear(in_features=2048, out_features=512, bias=False)\n",
            "          (o_proj): Linear(in_features=2048, out_features=2048, bias=False)\n",
            "          (rotary_emb): LlamaRotaryEmbedding()\n",
            "        )\n",
            "        (mlp): LlamaMLP(\n",
            "          (gate_proj): Linear(in_features=2048, out_features=6554, bias=False)\n",
            "          (up_proj): Linear(in_features=2048, out_features=6554, bias=False)\n",
            "          (down_proj): Linear(in_features=6554, out_features=2048, bias=False)\n",
            "          (act_fn): SiLU()\n",
            "        )\n",
            "        (input_layernorm): LlamaRMSNorm((2048,), eps=1e-05)\n",
            "        (post_attention_layernorm): LlamaRMSNorm((2048,), eps=1e-05)\n",
            "      )\n",
            "    )\n",
            "    (norm): LlamaRMSNorm((2048,), eps=1e-05)\n",
            "    (rotary_emb): LlamaRotaryEmbedding()\n",
            "  )\n",
            "  (lm_head): Linear(in_features=2048, out_features=128256, bias=False)\n",
            ")\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "In the model’s structure, at first glance, there doesn’t seem to be any error, but the MLP block structure has not been properly considered.\n",
        "\n",
        "The layers are being treated as if they were from the distilGPT model, whereas Llama uses a `GLU (Gated Linear Unit)` structure, where the `gate_proj` and `up_proj` layers work together. Therefore, pruning cannot be done by calculating the importance of neurons separately and removing different neurons in each layer. Instead, the pruning process should respect that these layers function as pairs.\n",
        "\n",
        "Thus, the evaluation of which neurons to prune should take into account that they must be assessed together, and pruning should be done on pairs of neurons.\n",
        "\n",
        "We now have some key points that need to be addressed in order to develop a pruning solution that suits the Llama model’s structure.\n",
        "\n",
        "* Consider the GLU (Gated Linear Unit) structure of the MLP layers.\n",
        "* Use a neuron selection method that is compatible with the GLU structure.\n",
        "\n",
        "**We will explore this in the next notebook.**"
      ],
      "metadata": {
        "id": "PfxxChCWXrco"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Upload the model to Hugging Face & Download to test.\n",
        "\n",
        "Even if the model isn't fully functional, let's at least check that it can work properly with the Transformers library."
      ],
      "metadata": {
        "id": "myfZBi1wktTo"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "output_dir = './pruned_llama_1b'\n",
        "if not os.path.exists(output_dir):\n",
        "    os.makedirs(output_dir)\n",
        "\n",
        "model.save_pretrained(output_dir)\n",
        "tokenizer.save_pretrained(output_dir)\n",
        "print(f\"Pruned model saved to {output_dir}\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "S2Ll_kqe5QzO",
        "outputId": "a3a63572-5e21-40ed-8498-014420e29523"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Pruned model saved to ./pruned_llama_1b\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Push the model to your Hugging Face repository\n",
        "model.push_to_hub('pruned-llama-1b')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 120,
          "referenced_widgets": [
            "cece5695569b4532a42a0357874653f3",
            "722a7745cc854eed957e6d0e0504ab9f",
            "a222a90b5f4a4b2588ba94fabc213c6f",
            "ba3630e5a35e419facbcfeffd2c6e3b5",
            "d9501a0de69642a983a997440cc503e9",
            "79e115b1505f45d593ca88f4918edb48",
            "0f7935a9542e4e7fbcc3281b09d031d9",
            "d8c41ad13145411fb8d2f9f5022a9922",
            "ae5ab815572e4c139c0adc8cec2d6f3f",
            "275deb9c92f4446a9e8bcc10f6acc598",
            "d83f182045164f0e9e3dc5c692135339"
          ]
        },
        "id": "3LjjsGZV5ZHJ",
        "outputId": "a0068387-3097-42de-bbf6-606d74ad4e02"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "README.md:   0%|          | 0.00/5.17k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "cece5695569b4532a42a0357874653f3"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "No files have been modified since last commit. Skipping to prevent empty commit.\n",
            "WARNING:huggingface_hub.hf_api:No files have been modified since last commit. Skipping to prevent empty commit.\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "CommitInfo(commit_url='https://huggingface.co/oopere/pruned-llama-1b/commit/383b3b2cf8bec7bb7df853261150ee448cc67757', commit_message='Upload LlamaForCausalLM', commit_description='', oid='383b3b2cf8bec7bb7df853261150ee448cc67757', pr_url=None, pr_revision=None, pr_num=None)"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 31
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "tokenizer.push_to_hub('pruned-llama-1b')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 88
        },
        "id": "6xNN-aYa5h9B",
        "outputId": "3e923716-6c27-4825-ee46-5be36f10e642"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "No files have been modified since last commit. Skipping to prevent empty commit.\n",
            "WARNING:huggingface_hub.hf_api:No files have been modified since last commit. Skipping to prevent empty commit.\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "CommitInfo(commit_url='https://huggingface.co/oopere/pruned-llama-1b/commit/383b3b2cf8bec7bb7df853261150ee448cc67757', commit_message='Upload tokenizer', commit_description='', oid='383b3b2cf8bec7bb7df853261150ee448cc67757', pr_url=None, pr_revision=None, pr_num=None)"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 32
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Download the pruned model\n",
        "pruned_model_name = 'oopere/pruned-llama-1b'\n",
        "pruned_model = AutoModelForCausalLM.from_pretrained(pruned_model_name, torch_dtype=torch.float16).to(device)\n",
        "pruned_tokenizer = AutoTokenizer.from_pretrained(pruned_model_name)\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 209,
          "referenced_widgets": [
            "20a5b71d4bf941ecb0658807d499987b",
            "90c4ba92707f45d8913cea121aecb787",
            "1953202045d44bfeb69c48c5539a7ef0",
            "fb3e7a68c9314072bfbcd97d53c35501",
            "db45ad2e736443439a9f9f831678ad96",
            "d000d18a94954853a36989b06232a6e0",
            "2a2d0e5bb41c444cb51709f6b87311d1",
            "ccbcd291c3dd42ad8d403a306a72acaa",
            "2a3f6d7d3d9f40afa0e0b0864207f0b2",
            "460b91727b4e40028886c84a1f2939d3",
            "7c77de5450554cb3b660348ce5e49a0e",
            "a559fdc00e2f418a9271507fce253084",
            "d376c161c90041c4b52ba2280d4318d2",
            "16a9ccc207e243fcbc0767bb074a6c0d",
            "c55644804f1543339db4ca36496e605a",
            "303a92373583484ab1a4b99adeccbdc2",
            "d469f4c9ba2c4d21bbe7b80f116bf6c3",
            "d5c0cf067bad4b2f9bcfbd68599511a6",
            "b584051809f44f0282c066758b7f0b02",
            "8c14bb5122334c4c89df1057a2052bab",
            "8309f0207309403eb8c352c6abc13b8f",
            "e0a333ba28c1468398dee7e31b31ca03",
            "58606e01c9974bd0aab0c4c1acf5622c",
            "213c320fd8e14840a47b29684e28bcf7",
            "58cf861e60a7428fbf77946fa4305673",
            "8c1665efbb83429fa656424c6d5fea0d",
            "b31bb5958c6743f3a4a57112b42f4120",
            "a077359ffe4643d99e8863f43dddcc45",
            "20e3cfd7180e477d9ec300cc3d13f071",
            "5cdb91d8ab904faeaa8085926286d999",
            "76cf1894f25c4292ba893aa96a014f3c",
            "ed96a554823b403c9456def6de94e9db",
            "adbfb2fed0f5429bb4c2f055f4c60200",
            "63774101422c4bfdb38142eb7715e1be",
            "02c9cde98ece48b4adc2f20222b17ebb",
            "f0e33cbe39f04935b155f908a5e75794",
            "c45fb49784b645caa8dfa9411cddc1c7",
            "a324b4e0bb6c422fb80fbb8e8e72172a",
            "21dbb00986654628848fd8b287fcff9d",
            "35ad7f0a261d4cfc847c36f8398809a7",
            "f550089dca164f13943c12d6479c536f",
            "734dd78c1fb049ad94b743626016f06e",
            "ac642c1ced614456b02a5f39b74c83fa",
            "d488fd2e3d77411ca2a089e9f3492a26",
            "b356ae4d6f7f4dc78daae9f00a338e1b",
            "8eb3c7f1f7794a17aa79843468b3926c",
            "167b8ec730b046869bfeae60558eb9f8",
            "eee14d76c00b4ba5a7129eae083ee33d",
            "e29b9bdf5ca14faba8f13aeb4307deb8",
            "da0e1f16e0ec45d1acb710dcc9434bb9",
            "f76d13949c1b41c9a7a48ef9349425f9",
            "271b7314836d41a4947fc8749acffd76",
            "83b4b4a046264ad3a156019bb76192b3",
            "1b835b17d6fe4884b1a2e7e41bf988bb",
            "ae13d551207c40b3a4ca2ce873df5f52",
            "89a298045c2d4a239ce0cad2f3737eeb",
            "85d1c8ee730c40c9a8cac1fd152da378",
            "93a87cd1cfd24110be3403f8138ddff8",
            "11b0d4a003cb44fc8b7efef7022de0e0",
            "e4a7e41649674af291674b3f20ef3772",
            "777567f83624445cbc20bd9c965c0219",
            "22cbf68fb2b348ecb5bcd1e622158727",
            "80a266d0cb1447e3a0a7e207063d76f3",
            "bf15270bc3ff4344abc66a68be9ceac8",
            "c1e13124110844dd8ec749009492cbc3",
            "c345ff9ff45140ba9a92974873178d75"
          ]
        },
        "id": "81iZY5ru6B7h",
        "outputId": "88ad511b-797c-4762-edec-507ccbf1e20f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "config.json:   0%|          | 0.00/883 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "20a5b71d4bf941ecb0658807d499987b"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "model.safetensors:   0%|          | 0.00/2.15G [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "a559fdc00e2f418a9271507fce253084"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "generation_config.json:   0%|          | 0.00/180 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "58606e01c9974bd0aab0c4c1acf5622c"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "tokenizer_config.json:   0%|          | 0.00/50.5k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "63774101422c4bfdb38142eb7715e1be"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "tokenizer.json:   0%|          | 0.00/9.09M [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "b356ae4d6f7f4dc78daae9f00a338e1b"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "special_tokens_map.json:   0%|          | 0.00/335 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "89a298045c2d4a239ce0cad2f3737eeb"
            }
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Test the downloaded pruned model\n",
        "generated = get_output(prompt, pruned_model, pruned_tokenizer)\n",
        "print(f\"Generated text from downloaded pruned model: {generated}\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MqCTEMGq6E6i",
        "outputId": "683af90a-3ad8-4ab3-f1ab-1306ff4b35b1"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Generated text from downloaded pruned model: Paris is the capital of of of Europe. It is the the the the the largest city of France. It is the the the the the the the the the the the the the the the the the the the the the the the the the is\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Conclusion.\n",
        "This notebook reminds us that a pruning process must consider the model’s structure and cannot simply reuse the same approach for models with different architectures.\n",
        "\n",
        "What previously worked perfectly with the distilGPT2 model has rendered the Llama3.2 model completely unusable.\n",
        "\n",
        "## Future work.\n",
        "It's clear that the task for the next notebook will be to develop a pruning process that, while inspired by this one, is able to respect the model's structure and reduce its size without significantly impacting its functionality.\n",
        "\n",
        "##Authors Note.\n",
        "In addition to creating content like this notebook and offering it under the MIT license, I have also contributed to repositories such as those of Hugging Face and Google Gemini.\n",
        "\n",
        "I am especially proud of my book: <a href=\"https://amzn.to/4eanT1g\"><b>Large Language Models:</b> Apply and Implement Strategies for Large Language Models</a> (Apress).\n",
        "\n",
        "You can find it on both <a href=\"https://amzn.to/4eanT1g\">Amazon</a> and <a href=\"https://link.springer.com/book/10.1007/979-8-8688-0515-8\">Springer</a>, where they often have good deals on the purchase price.\n",
        "\n",
        "If you take a look and end up purchasing it, keep in mind that you can reach out with any questions via the Discussions section of this same repository or on any of my social media channels. I’ll do my best to respond as quickly as possible."
      ],
      "metadata": {
        "id": "Y1KC2x0CurG2"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "tEZXDfYoXG7m"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}